De-black-boxing health AI: demonstrating reproducible machine learning computable phenotypes using the N3C-RECOVER Long COVID model in the All of Us data repository.

Machine learning (ML)-driven computable phenotypes are among the most challenging to share and reproduce. Despite this difficulty, the urgent public health considerations around Long COVID make it especially important to ensure the rigor and reproducibility of Long COVID phenotyping algorithms such that they can be made available to a broad audience of researchers. As part of the NIH Researching COVID to Enhance Recovery (RECOVER) Initiative, researchers with the National COVID Cohort Collaborative (N3C) devised and trained an ML-based phenotype to identify patients highly probable to have Long COVID. Supported by RECOVER, N3C and NIH's All of Us study partnered to reproduce the output of N3C's trained model in the All of Us data enclave, demonstrating model extensibility in multiple environments. This case study in ML-based phenotype reuse illustrates how open-source software best practices and cross-site collaboration can de-black-box phenotyping algorithms, prevent unnecessary rework, and promote open science in informatics.

Risk of post-acute sequelae of SARS-CoV-2 infection associated with pre-coronavirus disease obstructive sleep apnea diagnoses: an electronic health record-based analysis from the RECOVER initiative.

STUDY OBJECTIVES: Obstructive sleep apnea (OSA) has been associated with more severe acute coronavirus disease-2019 (COVID-19) outcomes. We assessed OSA as a potential risk factor for Post-Acute Sequelae of SARS-CoV-2 (PASC). METHODS: We assessed the impact of preexisting OSA on the risk for probable PASC in adults and children using electronic health record data from multiple research networks. Three research networks within the REsearching COVID to Enhance Recovery initiative (PCORnet Adult, PCORnet Pediatric, and the National COVID Cohort Collaborative [N3C]) employed a harmonized analytic approach to examine the risk of probable PASC in COVID-19-positive patients with and without a diagnosis of OSA prior to pandemic onset. Unadjusted odds ratios (ORs) were calculated as well as ORs adjusted for age group, sex, race/ethnicity, hospitalization status, obesity, and preexisting comorbidities. RESULTS: Across networks, the unadjusted OR for probable PASC associated with a preexisting OSA diagnosis in adults and children ranged from 1.41 to 3.93. Adjusted analyses found an attenuated association that remained significant among adults only. Multiple sensitivity analyses with expanded inclusion criteria and covariates yielded results consistent with the primary analysis. CONCLUSIONS: Adults with preexisting OSA were found to have significantly elevated odds of probable PASC. This finding was consistent across data sources, approaches for identifying COVID-19-positive patients, and definitions of PASC. Patients with OSA may be at elevated risk for PASC after SARS-CoV-2 infection and should be monitored for post-acute sequelae.

Data-driven analysis to understand long COVID using electronic health records from the RECOVER initiative.

Recent studies have investigated post-acute sequelae of SARS-CoV-2 infection (PASC, or long COVID) using real-world patient data such as electronic health records (EHR). Prior studies have typically been conducted on patient cohorts with specific patient populations which makes their generalizability unclear. This study aims to characterize PASC using the EHR data warehouses from two large Patient-Centered Clinical Research Networks (PCORnet), INSIGHT and OneFlorida+, which include 11 million patients in New York City (NYC) area and 16.8 million patients in Florida respectively. With a high-throughput screening pipeline based on propensity score and inverse probability of treatment weighting, we identified a broad list of diagnoses and medications which exhibited significantly higher incidence risk for patients 30-180 days after the laboratory-confirmed SARS-CoV-2 infection compared to non-infected patients. We identified more PASC diagnoses in NYC than in Florida regarding our screening criteria, and conditions including dementia, hair loss, pressure ulcers, pulmonary fibrosis, dyspnea, pulmonary embolism, chest pain, abnormal heartbeat, malaise, and fatigue, were replicated across both cohorts. Our analyses highlight potentially heterogeneous risks of PASC in different populations.

Racial/Ethnic Disparities in Post-acute Sequelae of SARS-CoV-2 Infection in New York: an EHR-Based Cohort Study from the RECOVER Program.

BACKGROUND: Compared to white individuals, Black and Hispanic individuals have higher rates of COVID-19 hospitalization and death. Less is known about racial/ethnic differences in post-acute sequelae of SARS-CoV-2 infection (PASC). OBJECTIVE: Examine racial/ethnic differences in potential PASC symptoms and conditions among hospitalized and non-hospitalized COVID-19 patients. DESIGN: Retrospective cohort study using data from electronic health records. PARTICIPANTS: 62,339 patients with COVID-19 and 247,881 patients without COVID-19 in New York City between March 2020 and October 2021. MAIN MEASURES: New symptoms and conditions 31-180 days after COVID-19 diagnosis. KEY RESULTS: The final study population included 29,331 white patients (47.1%), 12,638 Black patients (20.3%), and 20,370 Hispanic patients (32.7%) diagnosed with COVID-19. After adjusting for confounders, significant racial/ethnic differences in incident symptoms and conditions existed among both hospitalized and non-hospitalized patients. For example, 31-180 days after a positive SARS-CoV-2 test, hospitalized Black patients had higher odds of being diagnosed with diabetes (adjusted odds ratio [OR]: 1.96, 95% confidence interval [CI]: 1.50-2.56, q<0.001) and headaches (OR: 1.52, 95% CI: 1.11-2.08, q=0.02), compared to hospitalized white patients. Hospitalized Hispanic patients had higher odds of headaches (OR: 1.62, 95% CI: 1.21-2.17, q=0.003) and dyspnea (OR: 1.22, 95% CI: 1.05-1.42, q=0.02), compared to hospitalized white patients. Among non-hospitalized patients, Black patients had higher odds of being diagnosed with pulmonary embolism (OR: 1.68, 95% CI: 1.20-2.36, q=0.009) and diabetes (OR: 2.13, 95% CI: 1.75-2.58, q<0.001), but lower odds of encephalopathy (OR: 0.58, 95% CI: 0.45-0.75, q<0.001), compared to white patients. Hispanic patients had higher odds of being diagnosed with headaches (OR: 1.41, 95% CI: 1.24-1.60, q<0.001) and chest pain (OR: 1.50, 95% CI: 1.35-1.67, q < 0.001), but lower odds of encephalopathy (OR: 0.64, 95% CI: 0.51-0.80, q<0.001). CONCLUSIONS: Compared to white patients, patients from racial/ethnic minority groups had significantly different odds of developing potential PASC symptoms and conditions. Future research should examine the reasons for these differences.

Identifying environmental risk factors for post-acute sequelae of SARS-CoV-2 infection: An EHR-based cohort study from the recover program.

Post-acute sequelae of SARS-CoV-2 infection (PASC) affects a wide range of organ systems among a large proportion of patients with SARS-CoV-2 infection. Although studies have identified a broad set of patient-level risk factors for PASC, little is known about the association between "exposome"-the totality of environmental exposures and the risk of PASC. Using electronic health data of patients with COVID-19 from two large clinical research networks in New York City and Florida, we identified environmental risk factors for 23 PASC symptoms and conditions from nearly 200 exposome factors. The three domains of exposome include natural environment, built environment, and social environment. We conducted a two-phase environment-wide association study. In Phase 1, we ran a mixed effects logistic regression with 5-digit ZIP Code tabulation area (ZCTA5) random intercepts for each PASC outcome and each exposome factor, adjusting for a comprehensive set of patient-level confounders. In Phase 2, we ran a mixed effects logistic regression for each PASC outcome including all significant (false positive discovery adjusted p-value < 0.05) exposome characteristics identified from Phase I and adjusting for confounders. We identified air toxicants (e.g., methyl methacrylate), particulate matter (PM2.5) compositions (e.g., ammonium), neighborhood deprivation, and built environment (e.g., food access) that were associated with increased risk of PASC conditions related to nervous, blood, circulatory, endocrine, and other organ systems. Specific environmental risk factors for each PASC condition and symptom were different across the New York City area and Florida. Future research is warranted to extend the analyses to other regions and examine more granular exposome characteristics to inform public health efforts to help patients recover from SARS-CoV-2 infection.

Data-driven identification of post-acute SARS-CoV-2 infection subphenotypes.

The post-acute sequelae of SARS-CoV-2 infection (PASC) refers to a broad spectrum of symptoms and signs that are persistent, exacerbated or newly incident in the period after acute SARS-CoV-2 infection. Most studies have examined these conditions individually without providing evidence on co-occurring conditions. In this study, we leveraged the electronic health record data of two large cohorts, INSIGHT and OneFlorida+, from the national Patient-Centered Clinical Research Network. We created a development cohort from INSIGHT and a validation cohort from OneFlorida+ including 20,881 and 13,724 patients, respectively, who were SARS-CoV-2 infected, and we investigated their newly incident diagnoses 30-180 days after a documented SARS-CoV-2 infection. Through machine learning analysis of over 137 symptoms and conditions, we identified four reproducible PASC subphenotypes, dominated by cardiac and renal (including 33.75% and 25.43% of the patients in the development and validation cohorts); respiratory, sleep and anxiety (32.75% and 38.48%); musculoskeletal and nervous system (23.37% and 23.35%); and digestive and respiratory system (10.14% and 12.74%) sequelae. These subphenotypes were associated with distinct patient demographics, underlying conditions before SARS-CoV-2 infection and acute infection phase severity. Our study provides insights into the heterogeneity of PASC and may inform stratified decision-making in the management of PASC conditions.

Influence of social deprivation index on in-hospital outcomes of COVID-19.

While it is known that social deprivation index (SDI) plays an important role on risk for acquiring Coronavirus Disease 2019 (COVID-19), the impact of SDI on in-hospital outcomes such as intubation and mortality are less well-characterized. We analyzed electronic health record data of adults hospitalized with confirmed COVID-19 between March 1, 2020 and February 8, 2021 from the INSIGHT Clinical Research Network (CRN). To compute the SDI (exposure variable), we linked clinical data using patient's residential zip-code with social data at zip-code tabulation area. SDI is a composite of seven socioeconomic characteristics determinants at the zip-code level. For this analysis, we categorized SDI into quintiles. The two outcomes of interest were in-hospital intubation and mortality. For each outcome, we examined logistic regression and random forests to determine incremental value of SDI in predicting outcomes. We studied 30,016 included COVID-19 patients. In a logistic regression model for intubation, a model including demographics, comorbidity, and vitals had an Area under the receiver operating characteristic curve (AUROC) = 0.73 (95% CI 0.70-0.75); the addition of SDI did not improve prediction [AUROC = 0.73 (95% CI 0.71-0.75)]. In a logistic regression model for in-hospital mortality, demographics, comorbidity, and vitals had an AUROC = 0.80 (95% CI 0.79-0.82); the addition of SDI in Model 2 did not improve prediction [AUROC = 0.81 (95% CI 0.79-0.82)]. Random forests revealed similar findings. SDI did not provide incremental improvement in predicting in-hospital intubation or mortality. SDI plays an important role on who acquires COVID-19 and its severity; but once hospitalized, SDI appears less important.

Racial and Ethnic Disparities in Receipt of Medications for Treatment of COVID-19 - United States, March 2020-August 2021.

The COVID-19 pandemic has magnified longstanding health care and social inequities, resulting in disproportionately high COVID-19-associated illness and death among members of racial and ethnic minority groups (1). Equitable use of effective medications (2) could reduce disparities in these severe outcomes (3). Monoclonal antibody (mAb) therapies against SARS-CoV-2, the virus that causes COVID-19, initially received Emergency Use Authorization (EUA) from the Food and Drug Administration (FDA) in November 2020. mAbs are typically administered in an outpatient setting via intravenous infusion or subcutaneous injection and can prevent progression of COVID-19 if given after a positive SARS-CoV-2 test result or for postexposure prophylaxis in patients at high risk for severe illness.† Dexamethasone, a commonly used steroid, and remdesivir, an antiviral drug that received EUA from FDA in May 2020, are used in inpatient settings and help prevent COVID-19 progression§ (2). No large-scale studies have yet examined the use of mAb by race and ethnicity. Using COVID-19 patient electronic health record data from 41 U.S. health care systems that participated in the PCORnet, the National Patient-Centered Clinical Research Network,¶ this study assessed receipt of medications for COVID-19 treatment by race (White, Black, Asian, and Other races [including American Indian or Alaska Native, Native Hawaiian or Other Pacific Islander, and multiple or Other races]) and ethnicity (Hispanic or non-Hispanic). Relative disparities in mAb** treatment among all patients†† (805,276) with a positive SARS-CoV-2 test result and in dexamethasone and remdesivir treatment among inpatients§§ (120,204) with a positive SARS-CoV-2 test result were calculated. Among all patients with positive SARS-CoV-2 test results, the overall use of mAb was infrequent, with mean monthly use at 4% or less for all racial and ethnic groups. Hispanic patients received mAb 58% less often than did non-Hispanic patients, and Black, Asian, or Other race patients received mAb 22%, 48%, and 47% less often, respectively, than did White patients during November 2020-August 2021. Among inpatients, disparities were different and of lesser magnitude: Hispanic inpatients received dexamethasone 6% less often than did non-Hispanic inpatients, and Black inpatients received remdesivir 9% more often than did White inpatients. Vaccines and preventive measures are the best defense against infection; use of COVID-19 medications postexposure or postinfection can reduce morbidity and mortality and relieve strain on hospitals but are not a substitute for COVID-19 vaccination. Public health policies and programs centered around the specific needs of communities can promote health equity (4). Equitable receipt of outpatient treatments, such as mAb and antiviral medications, and implementation of prevention practices are essential to reducing existing racial and ethnic inequities in severe COVID-19-associated illness and death.

Comparing automated vs. manual data collection for COVID-specific medications from electronic health records.

INTRODUCTION: Data extraction from electronic health record (EHR) systems occurs through manual abstraction, automated extraction, or a combination of both. While each method has its strengths and weaknesses, both are necessary for retrospective observational research as well as sudden clinical events, like the COVID-19 pandemic. Assessing the strengths, weaknesses, and potentials of these methods is important to continue to understand optimal approaches to extracting clinical data. We set out to assess automated and manual techniques for collecting medication use data in patients with COVID-19 to inform future observational studies that extract data from the electronic health record (EHR). MATERIALS AND METHODS: For 4,123 COVID-positive patients hospitalized and/or seen in the emergency department at an academic medical center between 03/03/2020 and 05/15/2020, we compared medication use data of 25 medications or drug classes collected through manual abstraction and automated extraction from the EHR. Quantitatively, we assessed concordance using Cohen's kappa to measure interrater reliability, and qualitatively, we audited observed discrepancies to determine causes of inconsistencies. RESULTS: For the 16 inpatient medications, 11 (69%) demonstrated moderate or better agreement; 7 of those demonstrated strong or almost perfect agreement. For 9 outpatient medications, 3 (33%) demonstrated moderate agreement, but none achieved strong or almost perfect agreement. We audited 12% of all discrepancies (716/5,790) and, in those audited, observed three principal categories of error: human error in manual abstraction (26%), errors in the extract-transform-load (ETL) or mapping of the automated extraction (41%), and abstraction-query mismatch (33%). CONCLUSION: Our findings suggest many inpatient medications can be collected reliably through automated extraction, especially when abstraction instructions are designed with data architecture in mind. We discuss quality issues, concerns, and improvements for institutions to consider when crafting an approach. During crises, institutions must decide how to allocate limited resources. We show that automated extraction of medications is feasible and make recommendations on how to improve future iterations.

Socioeconomic variation in characteristics, outcomes, and healthcare utilization of COVID-19 patients in New York City.

OBJECTIVES: There is limited evidence on how clinical outcomes differ by socioeconomic conditions among patients with coronavirus disease 2019 (COVID-19). Most studies focused on COVID-19 patients from a single hospital. Results based on patients from multiple health systems have not been reported. The objective of this study is to examine variation in patient characteristics, outcomes, and healthcare utilization by neighborhood social conditions among COVID-19 patients. METHODS: We extracted electronic health record data for 23,300 community dwelling COVID-19 patients in New York City between March 1st and June 11th, 2020 from all care settings, including hospitalized patients, patients who presented to the emergency department without hospitalization, and patients with ambulatory visits only. Zip Code Tabulation Area-level social conditions were measured by the Social Deprivation Index (SDI). Using logistic regressions and Cox proportional-hazards models, we examined the association between SDI quintiles and hospitalization and death, controlling for race, ethnicity, and other patient characteristics. RESULTS: Among 23,300 community dwelling COVID-19 patients, 60.7% were from neighborhoods with disadvantaged social conditions (top SDI quintile), although these neighborhoods only account for 34% of overall population. Compared to socially advantaged patients (bottom SDI quintile), socially disadvantaged patients (top SDI quintile) were older (median age 55 vs. 53, P<0.001), more likely to be black (23.1% vs. 6.4%, P<0.001) or Hispanic (25.4% vs. 8.5%, P<0.001), and more likely to have chronic conditions (e.g., diabetes: 21.9% vs. 10.5%, P<0.001). Logistic and Cox regressions showed that patients with disadvantaged social conditions had higher risk for hospitalization (odds ratio: 1.68; 95% confidence interval [CI]: [1.46, 1.94]; P<0.001) and mortality (hazard ratio: 1.91; 95% CI: [1.35, 2.70]; P<0.001), adjusting for other patient characteristics. CONCLUSION: Substantial socioeconomic disparities in health outcomes exist among COVID-19 patients in NYC. Disadvantaged neighborhood social conditions were associated with higher risk for hospitalization, severity of disease, and death.

Clinical subphenotypes in COVID-19: derivation, validation, prediction, temporal patterns, and interaction with social determinants of health.

The coronavirus disease 2019 (COVID-19) is heterogeneous and our understanding of the biological mechanisms of host response to the viral infection remains limited. Identification of meaningful clinical subphenotypes may benefit pathophysiological study, clinical practice, and clinical trials. Here, our aim was to derive and validate COVID-19 subphenotypes using machine learning and routinely collected clinical data, assess temporal patterns of these subphenotypes during the pandemic course, and examine their interaction with social determinants of health (SDoH). We retrospectively analyzed 14418 COVID-19 patients in five major medical centers in New York City (NYC), between March 1 and June 12, 2020. Using clustering analysis, 4 biologically distinct subphenotypes were derived in the development cohort (N = 8199). Importantly, the identified subphenotypes were highly predictive of clinical outcomes (especially 60-day mortality). Sensitivity analyses in the development cohort, and rederivation and prediction in the internal (N = 3519) and external (N = 3519) validation cohorts confirmed the reproducibility and usability of the subphenotypes. Further analyses showed varying subphenotype prevalence across the peak of the outbreak in NYC. We also found that SDoH specifically influenced mortality outcome in Subphenotype IV, which is associated with older age, worse clinical manifestation, and high comorbidity burden. Our findings may lead to a better understanding of how COVID-19 causes disease in different populations and potentially benefit clinical trial development. The temporal patterns and SDoH implications of the subphenotypes may add insights to health policy to reduce social disparity in the pandemic.

A predictive model of clinical deterioration among hospitalized COVID-19 patients by harnessing hospital course trajectories.

From early March through mid-May 2020, the COVID-19 pandemic overwhelmed hospitals in New York City. In anticipation of ventilator shortages and limited ICU bed capacity, hospital operations prioritized the development of prognostic tools to predict clinical deterioration. However, early experience from frontline physicians observed that some patients developed unanticipated deterioration after having relatively stable periods, attesting to the uncertainty of clinical trajectories among hospitalized patients with COVID-19. Prediction tools that incorporate clinical variables at one time-point, usually on hospital presentation, are suboptimal for patients with dynamic changes and evolving clinical trajectories. Therefore, our study team developed a machine-learning algorithm to predict clinical deterioration among hospitalized COVID-19 patients by extracting clinically meaningful features from complex longitudinal laboratory and vital sign values during the early period of hospitalization with an emphasis on informative missing-ness. To incorporate the evolution of the disease and clinical practice over the course of the pandemic, we utilized a time-dependent cross-validation strategy for model development. Finally, we validated our prediction model on an external validation cohort of COVID-19 patients served in a demographically distinct population from the training cohort. The main finding of our study is the identification of risk profiles of early, late and no clinical deterioration during the course of hospitalization. While risk prediction models that include simple predictors at ED presentation and clinical judgement are able to identify any deterioration vs. no deterioration, our methodology is able to isolate a particular risk group that remain stable initially but deteriorate at a later stage of the course of hospitalization. We demonstrate the superior predictive performance with the utilization of laboratory and vital sign data during the early period of hospitalization compared to the utilization of data at presentation alone. Our results will allow efficient hospital resource allocation and will motivate research in understanding the late deterioration risk group.

Use of electronic health records to support a public health response to the COVID-19 pandemic in the United States: a perspective from 15 academic medical centers.

Our goal is to summarize the collective experience of 15 organizations in dealing with uncoordinated efforts that result in unnecessary delays in understanding, predicting, preparing for, containing, and mitigating the COVID-19 pandemic in the US. Response efforts involve the collection and analysis of data corresponding to healthcare organizations, public health departments, socioeconomic indicators, as well as additional signals collected directly from individuals and communities. We focused on electronic health record (EHR) data, since EHRs can be leveraged and scaled to improve clinical care, research, and to inform public health decision-making. We outline the current challenges in the data ecosystem and the technology infrastructure that are relevant to COVID-19, as witnessed in our 15 institutions. The infrastructure includes registries and clinical data networks to support population-level analyses. We propose a specific set of strategic next steps to increase interoperability, overall organization, and efficiencies.